Software products are oftentimes subjected to a variety of tests prior to and in the early stages of deployment. The number of tests that a software product is to go through depends on a variety of factors such as, for example, how large and/or complicated the program is, whether/how it relates to mission-critical or other key operations, the extent to which it interacts with other systems, etc. It is not uncommon for an enterprise product to be subjected to thousands of tests prior to deployment. For instance, during bug fixing or feature enhancement, a fairly typical design practice involves running (or re-running) all of the tests in the test plan to help validate (or re-validate) the application and make sure the new code change does not “break” any existing functionality.
One need not necessarily run all the tests at this time, at least in theory, because a given code change need not necessarily affect how all tests perform. In other words, some changes will have limited scopes and, as a result, it at least theoretically should be possible to avoid having to run (or re-run) the complete set of tests for all changes.
Unfortunately, however, it is not always readily apparent which code changes will affect the various parts of an application and/or the application's overall performance. Furthermore, complications in assessing the potential scope of a change may arise in the context of systems that are live, as it is not always easy to simply suspend operations for testing purposes. And yet further complications may arise in distributed environments.
U.S. Publication No. 2005/0223361 to Belbute, for example, unfortunately seems to involve a laborious process, e.g., as its realization seems to involve multiple products and also seems to involve instrumenting the code prior running the tests. Belbute also seems to rely on the order of nodes (lines of code). That is, the way it detects changes involves recording the node number it visits. Its approach therefore might not able to detect the change if a line of code is removed and another one is added at the same place. Moreover, it is unclear how changes at the class level are handled, as compared to changes at the method level. Its overall architecture might not be a good fit for distributed teams, in general. Belbute also seems to lack a learning mechanism such as, for example, a feedback loop.
U.S. Publication No. US 2011/0145793 to Alexander et al. apparently provides an approach for tagging components in a software application and assigning them to a test scheme. Unfortunately, as above, its process seems to be laborious. Because it uses code coverage (tools) as the mechanism to track changes, this needs to be done for each test case in a test suite. Thus, one cannot simply run the test suite. Instead, each test should be interrupted to allow code coverage to record and reset it values. This might seemingly imply a manual approach to certain aspects. And as with Belbute, Alexander seems to rely on the line numbers of the code, e.g., in connection with its code coverage tool visits, which is disadvantageous for at least the reasons identified above. The lack of a learning mechanism (e.g., a feedback loop) also potentially slows debugging and/or testing processes.
Thus, it will be appreciated that there is a need in the art for improved testing techniques, e.g., for pre- and/or post-deployed applications.
One aspect of certain example embodiments relates to techniques by which a software engineer or other individual can determine the most probable subset of possible tests that is sufficient to validate a code change, program modification, update, or the like. By executing the subset of appropriate tests, it advantageously becomes possible to improve productivity and save resources that otherwise might be wasted during the execution of inappropriate or unnecessary tests. Advantageously, certain example techniques described herein may be used in connection with distributed product development teams (e.g., teams comprising individuals and/or sharing processing resources located remote from one another).
Another aspect of certain example embodiments relates to an incremental but consistent way of analyzing performance issues.
In accordance with certain example embodiments, a first unique identifier is created for each method and/or other structure in a source file (and potentially for the source file itself) to be used in checking the integrity of the method when required. Tests are executed, and information is collected concerning the code structures (e.g., the methods) that each test cuts across. The time required to execute the tests also may be collected, and the failed tests that are to be fed-back into the system are recorded (e.g., the relevant tests for a code change may be executed, with previous failed tests being prioritized in the execution order.) A second unique identifier is created for each method and/or other structure in the modified source file (and potentially for the modified source file itself). If there is a difference between the first and second unique identifiers, then the information that was collected is used to arrive at a list of appropriate tests to be executed under different circumstances (e.g., for each test associated method).
Some source repositories may index and/or timestamp the code they include, since this oftentimes provides enough detail to identify the latest version of the code. Using an MD5 or other checksum, however, provides an identification technique that very likely distinguishes every changed method or other code structure from the next. Thus, it is possible to make the mapping (e.g., between the code structures and the tests that invoke or otherwise involve them) match to a particular version tested at a predetermined time, not to just the latest version.
In certain example embodiments, a method for testing computer code is provided. Source code related to an application to be executed on a computer system including processing resources is received. The source code is parsed into plural identified structures. Using a test computer, a hash algorithm is applied to each of the identified structures in the source code in order to generate a corresponding first unique identifier. The first unique identifiers are stored to a non-transitory computer readable storage location. Under the direction of the test computer, tests are performed on the application in connection with the source code. Using the test computer, structures in the source code associated with each test are identified. A mapping between the performed tests and the identified structures associated with the tests is stored to a or the non-transitory computer readable storage location.
According to certain example embodiments, in addition to some or all of these features, modified source code related to the application is received. The modified source code is parsed into plural identified structures. The hash algorithm is applied to each of the identified structures in the modified source code in order to generate a corresponding second unique identifier. First and second unique identifiers related to corresponding identified structures in the source code and the modified source code, respectively, are compared. When a difference is detected between compared first and second unique identifiers, a list of any tests associated with the non-matching identifiers is generated from the mapping.
According to certain example embodiments, in addition to some or all of these features, first computer processing resource performance information is gathered for the processing resources of the computer system during execution of the application in connection with the source code. The first computer processing resource performance information is stored in connection with the mapping. Modified source code related to the application is received. Second computer processing resource performance information is gathered for the processing resources of the computer system during execution of the application in connection with the modified source code. First and second computer processing resource performance information is compared for each of the processing resources of the computer system. When a difference is detected between the compared first and second computer processing resource performance information and the difference exceeds a predetermined threshold value: the modified source code is parsed into plural identified structures; the hash algorithm is applied to each of the identified structures in the modified source code in order to generate a corresponding second unique identifier; first and second unique identifiers related to corresponding identified structures in the source code and the modified source code, respectively, are compared; and when a difference is detected between compared first and second unique identifiers, a list of any tests associated with the non-matching is generated from the mapping.
In certain example embodiments, non-transitory computer readable storage media tangibly storing instructions that, when executed by at least one processor of a computer, may perform one of these and/or other methods.
Identifiable structures may include method and non-method structures, in certain example embodiments, identifiable non-method structures possibly including class structures in some cases.
In certain example embodiments, a test system is provided. At least one processor and a memory are arranged to execute plural modules. For instance, a parser module is configured to receive source code related to an application to be executed on a computer system including processing resources and to identify plural structures in the received source code. The structures are identifiable as being one of a method structure or a non-method structure. A digester module is configured to apply a hash algorithm to each identified structure in the source code in order to generate a corresponding unique identifier. A non-transitory computer readable medium is configured to store the unique identifiers. The test system is configured to use its at least one processor to at least: cause tests to be performed on the application in connection with the source code, determine which identified structure(s) in the source code is/are associated with each said test, and store a mapping between the performed tests and the identified structures associated with the tests to a or the non-transitory computer readable medium.
According to certain example embodiments, in addition to some or all of these features, original source code and modified source code are provided to the parser module, and first identified structures for the original source code and second identified structures for the modified source code are provided to the digester module. A comparison module is configured to: compare first and second unique identifiers related to corresponding identified structures in the original source code and the modified source code, respectively, and generate from the mapping, when a difference is detected between compared first and second unique identifiers, a list of any tests associated with the non-matching identifiers.
According to certain example embodiments, in addition to some or all of these features, an application profiler is provided and is configured to track performance statistics of the processing resources of the computer system on which the application is executed, at least during testing. For instance, the application profiler is configured to track performance statistics of the processing resources of the computer system on which the application is executed during testing for both first and second separate builds of the application. A first comparison module is configured to compare performance statistics for the first and second builds, with code for the first and second builds being provided to the parser, and with identified structures generated by the parser being provided to the digester in order to generate first and second unique identifiers for the first and second builds, respectively. A second comparison module is configured to compare first and second unique identifiers related to corresponding identified structures in the first and second builds, respectively, when the first comparison module indicates that there is a potentially significant deviation between performance statistics associated with the first and second builds. A report module is configured to generate a list of any tests associated with the first and second unique identifiers that do not match as determined by the second comparison module.
According to certain example embodiments, in addition to some or all of these features, a listener is provided and is configured to detect code check-in events at a source code repository. For instance, the listener is further configured to initiate testing as between code designated as baseline code and newly checked-in code by: providing to the parser module the newly checked-in code; providing to the digester module second identified structures for the newly checked-in code; and initiating a comparison module. That comparison module may compare first and second unique identifiers related to corresponding identified structures in the baseline code and the newly checked-in code, respectively, and generate from the mapping, when a difference is detected between compared first and second unique identifiers, a list of any tests associated with the non-matching identifiers.
It will be appreciated that non-matching identifiers may be caused as a result of changes as between different versions or snapshots in time, deletion of a structure, addition of a structure, etc.
The hash algorithm may be the MD5 algorithm, and the first and second unique identifiers are MD5 digests, in certain example embodiments.
The same or different distributed caches may be used in connection with different example embodiments.
These features, aspects, advantages, and example embodiments may be used separately and/or applied in various combinations to achieve yet further embodiments of this invention.